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ABSTRACT 

The Test of Essential Learning and Literacy Skills 
(TELLS) has been administered to all Pennsylvania students in grades 
3, 5, and 8 since 1985. An analysis of results from the 1,505 public 
elementary schools that served both third and fifth graders between 
1986 and 1990 show that in 150 of these schools more than 50 percent 
of the students scored below the minimum skill level. The figures 
also indicated that no matter for what grades or years scores are 
analyzed, schools tend to be consistent in their performance ranking. 
The achievement differences among schools are almost completely 
predictable from the socioeconomic status (SES) indicators of the 
families being served by them. Simple solutions for raising the basic 
skill level of 'students from homes with low SES, such as raising 
teacher salaries or reducing pupil/teacher ratios, do not look 
promising. Higher salaries can increase applicant pool quality but 
will not necessarily raise teaching staff quality. Smaller classes 
can be helpful if instruction is designed to take advantage of it. 
Proposals for improvement include investing in preschool experiences 
that increase school readiness skills. Four tables illustrate the 
data analysis. (8 references) (EJS) 
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Executive Summary 



This paper examines five years of state test results 
(TELLS), focusing upon differences among Pennsylvania's 
elementary schools, particularly those schools with a 
majority of their students not mastering essential 
learning skills. 

The largest learning problems occur in about 10% of 
the schools, and those schools have been consistently low 
for the past five years. Making public the test results 
of low scoring schools has not solved the problem. 

The achievement differences among schools is almost 
completely predictable from the socio-economic 
differences of the families being served by these 
schools. This is not news, but the persistent, dominant 
importance of home factors must be confronted if the 
situation is to improve. 

Simple but costly solutions, such as raising teacher 
salaries or reducing pupil/teacher ratios, do not look 
promising if that is all that is done. In fact, the 
lower scoring schools currently tend to have slightly 
better paid teachers and slightly smaller pupil teacher 
ratios. 

Increasing teacher salaries could be helpful in 
increasing the quality of the thousands of new teachers 
to be hired in the '90's if valid teacher quality 
indicators were available and used by districts in that 
hiring. Higher salaries can increase the quality of the 
applicant pool, but that will not necessarily increase 
the quality of the teaching staff if teachers are hired 
for reasons other than their ability to teach. 

Smaller classes can be helpful if instruction is 
designed to take advantage of it, and if use of smaller 
classes was managed, and didn't just happen. Currently 
pupil/teacher ratios are largely unrelated to student 
needs or to student achievement growth. If it is business 
as usual, the possible positive achievement effect of 
smaller classes will continue to be elusive. 

These results illustrate why it has been so 
difficult for research to establish the student 
achievement benefits of higher per pupil spending, since 
most spending variation among our 500 school districts is 
due to differences in teacher salaries or pupil/teacher 
ratios. 
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Testing and School Improvement 
William W. Cooley 

As the debates rage about how to reform, restructure 
and retest the nation's schools, it seems useful to 
examine what can be said about these issues based upon 
actual data from one state's testing program. This 
paper examines the results of testing in Pennsylvania's 
elementary schools between 1986 and 1990, and seeks 
implications for some cf the current debates. The 
results are derived from the state database established 
as part of the Pennsylvania Educational Policy Studies. 

The student performance information is based upon a 
Test of Essential Learning and Literacy Skills (TELLS), 
which has been administered to all students in grades 3, 
5 and 8 since 1985. The indicator of school performance 
used here is the percent of the students in the school 
that score below the minimum cut-score established by the 
state. Included in these analyses are the 1,505 public 
elementary schools that served both 3rd and 5th graders 
between 1986 and 1990. 

Figure 1 illustrates how that school performance 
indicator distributes for grade 5 reading in 1989, for 
example. As seen there, most of the schools had fewer 
than 30% of their students score below the cut score, but 
about 150 schools had more than 50% of their students 
score below the minimum (the schools in the right hand 
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"tail" of the distribution). Similar results were found 
for math and for all five years of TELLS testing. This 
paper explores why some schools seem to do so poorly, why 
it is serious, and what might be done to improve things. 

It must be emphasized at the outset that this paper 
is looking at schools, not individual students. Thus 
the relationships reported here are among schools, not 
students. For example, schools with a large percentage 
of students from homes on welfare tend to have large 
numbers of students performing low on the TELLS test. 
That school level relationship is significantly stronger 
than is the student level relationship between home 
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status and how individual students perform on the TELLS 
test. That is, it is much easier to predict the TELLS 
performance for a school than for a student, given home 
backgrounds. This point is important when considering 
what can be said about the effectiveness of a school , 
given school achievement results. 

It is also important to justify the school 
performance measure used here, which is the percent of 
students in the school who scored below the minimum cut 
score. The TELLS questions were the essential learning 
and literacy skills in reading and mathematics which 
committees of educators agreed all students should have 
mastered by that grade level. A student was below the 
cut score if more than about one-third of the questions 
were answered incorrectly. Even if one believes that the 
cut scores are arbitrary, that does not invalidate the 
kind of correlational analyses reported here. Changes in 
the cut score does not change the relative rankings of 
the schools. In fact, the relationships among schools 
based upon these established cut scores are identical to 
those that are obtained using school means as the school 
performance measure. 

Students have great difficulty learning from 
subsequent schooling if these basic skills are not 
mastered. For example, fourth grade instruction assumes 
the students possess the grade 3 TELLS skills. High 
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schools assume these skills have been mastered. 
Employers require them. Having an entire school at risk 
because most of the students in that school do not know 
these skills is not acceptable. 

Other papers in this PEPS series have shown why the 
TELLS test should be replaced by a more appropriate state 
test for purposes of accountability, curriculum reform, 
and informing state policy (Cooley & Bernauer, 1990 and 
Cooley, 1990). For example, TELLS measures too narrow a 
spectrum of curriculum goals to be the basis for 
determining what is to be taught. But the available 
TELLS results do allow us to examine several important 
current issues, and it seems reasonable to figure out 
what can be learned from the available test results 
before moving on to some new enthusiasm. People often 
call for a new test because we didn't learn much from the 
last test, but the implications of test results are never 
self evident, and we will continue to learn little or 
nothing from them if little or no analyses are conducted. 
TELLS does measure one aspect of what schooling is about, 
so let's see what we can learn. 
Stability of Schoo l Level Results 

The degree to which a school would be similarly 
ranked from one testing to another can best be described 
using correlations, where a correlation of 1.00 indicates 
identical rank orderings. Table 1 shows that school 
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Table 1 

Correlation Between Reading and Math 



Year 



Grade 3 



Grade 5 



1966 
1987 
1988 
1989 
1990 



.87 
.86 
.90 
.88 
.92 



.86 
.86 
.88 
.88 
.87 



Average • .88 



N - 1505 Schools 



rankings on student performance in reading and math are 
very similar for grades 3 and 5, and for all five years. 
That is, no matter what grade or year one examines, 
schools tend to have the same rank order whether one 
ranks reading or ranks math results. The average 
correlation in Table 1 is .88, with a range of .86 to 
.92, a very stable set of relationships. 

Table 2 reveals another remarkable stability in 
school performance. The correlations reported there show 
the stability of school performance between 3rd and 5th 
grade. For example, third gr iders in spring 1986 
represent the same cohort of stuc ents that were in grade 
5 in 1988, and reading performance at grade 3 correlates 
.82 with reading performance at grade 5. Therefore most 
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of the variation 
in school 
differences at 
grade 5 is 
predictable from 
their 
differences at 
grade 3. Notice 
that reading is 



Table 2 

Correlations for Same Cohort 

Year of 



Grade 


3 Grade 5 


Read 


Math 


1986 


1938 


.82 


.72 


1987 


1989 


.85 


.75 


1988 


1990 


.81 


.68 



more stable than math, but that both are very stable for 
the three cohorts available. 
School Performance and the Home 

Another indicator that is available for each school 
is the percent of the students that are from low income 
homes, particularly from families on welfare. Table 3 
reports the high correlation that exits between the 
percent of students below the TELLS cut-score and the 
percent of the students from low income families, for 
both grades for the five years. The average correlation 
for reading is .73, and for math it is .63. The square 
of the reading correlation indicates that 53% of the 
variation in reading performance among these 1,505 
schools is associated with this home socio-economic 
status (SES) indicator, and 40% for math. That is, 
reading performance is more dependent upon home factors 
than is math. 
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Table 3 

% Below Cut and % Low Income 



Year 


Grade 


Read 


Math 


1986 


3 


.69 


.60 




5 


.73 


.62 


1987 


3 


.71 


.61 




5 


.74 


.60 


1988 


3 


.70 


.62 




5 


.73 


.61 


1989 


3 


.76 


.65 




5 


.74 


.63 


1990 


3 


.74 


.70 




5 


.76 


.66 


ge Correlation 


.73 


.63 



N - 1505 Schools 



There are a few factors that affect the strength of 
the Table 3 relationships that should be mentioned. One 
such factor is the degree of overlap between the specific 
items in the TELLS test and the specifics of what was 
taught in each of these schools. As this overlap 
decreases, success on the test depends more and more on 
what is learned outside of school. Since there is no 
standard state curriculum, the resulting variation in 
overlap makes test performance more dependent upon home 
factors. This helps to explain why reading skills are 
more dependent upon home differences than are math 
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skills, since the latter are less likely to be learned 
outside of school. 

Also, the difficulty of a test affects its 
relationship with home factors. As more difficult items 
have been added to the TELLS test over time, there is 
better discrimination among schools with more able 
students, with correlations increasing slightly over 
time, as seen in Table 3. 

A third factor that affects the home-achievement 
relationship is the degree to which schools are 
homogeneous with respect to home SES. The more schools 
differ in their average demographic makeup, the higher 
will be the relationship between SES and school 
achievement. If, for example, schools were completely 
integrated with respect to SES, the strong relationships 
of Table 3 would no** be found. 

The very slight differences in these correlations 
within each column of Table 3 does reveal a high degree 
of consistency from year to year. It certainly shows 
that these indicators are not full of random error from 
grade to grade or year to year. But it also seems 
important to try to understand the implications of these 
trends for state testing programs and for school reform. 

One issue is whether the home influence persists as 
students move up the grades, or whether home SES simply 
affects the school readiness skills and other 
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predispositions with which students begin school. This 
question was examined using a technique called structural 
modeling, which allows one to examine whether the SES 
factor is relevant to 5th grade achievement, given their 
third grade performance. This analysis was possible 
because the PEPS data base included longitudinal data 
that were based upon the same students who were 3rd 
graders in 1987 and fifth graders in 1989. 

Structural Model 

SES and School Achievement 




Low Income 

i 

Figure 2 

Figure 2 illustrates a simple structural model. As 
indicated there, the SES factor is measured by the 
percent minority and the percent low income in the 
school. Grade 3 is a school-achievement-deficits factor 
indicating the degree to which students are scoring below 
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minimum on both reading and math. The Grade 5 factor 
indicates the degree to which this was true for these 
same students in fifth grade. The SES factor correlates 
.91 with Grade 3 and .94 with Grade 5. Grade 3 
correlates .86 with Grade 5. The main finding is that 
the SES factor continues to have a very strong direct 
effect between grade 3 and grade 5 school achievement. 
This means t'.iat whatever those processes are that result 
in strong relationships between SES and school 
performance, they continue as students move up the 
grades. Put another way, school differences in 5th grade 
achievement are even more dependent upon home differences 
than they were at 3rd grade. [The technical details of 
the structural modeling of these school level data will 
be provided in a forthcoming PEPS report.] 

Another sobering aspect of the Figure 2 analysis is 
the fact that this simple model explains 90% of the 
variation in 5th grade school achievement differences. 
That means that only 10% is left for all the different 
ways in which schools might be different: large or small, 
an effective or ineffective principal, curriculum similar 
to or different from the test, or the hundreds of other 
innovations, model schools, etc. that are out there. 

This does not mean that the situation is hopeless 
for schools serving low SES students, but it makes it 
very clear that the problem is not going to solved by 
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giving more testa. The 150 schools that were "in deep 
trouble" in 1986 are pretty much the same schools that 
are scoring low on the test today. Five years of public 
accountability (e.g., publishing school results in the 
local newspapers) has not made a difference. It makes it 
clear that reducing the large number of students not 
mastering basic skills in some schools is going to be 
very difficult. 

It is possible in this type of analysis to give 
homes undo credit or blame. Better teachers may be more 
likely, or good school practices more likely, in schools 
serving students from higher SES homes. If school 
practices or resources are correlated with home SES, then 
the Figure 2 analysis would be unfairly attributing to 
homes some of the credit or blame that was due the 
schools . 

Another PEPS report (Cooley, 1990) established that 
the major ways in which high-expenditure-per-pupil 
districts differed from low spenders is that they paid 
teachers more and hired more teachers relative to their 
student enrollment. So let's next examine how these 
schools differed in pupil/teacher ratio (P/T) and average 
teacher salary. Table 4 summarizes these correlations. 

For example, looking down the first column of Table 
4, Read3 is the percent of students below cut on the 
grade 3 TELLS reading test for 1987. That variable 

ERIC 1 4 
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correlates .85 with 1989 grade 5 reading, as was shown in 
Table 2, and correlates .69 with percent low income, as 
in Table 3. The percent minority correlates .72 with 
Read3, about the same as percent low income. The .13 
with teacher salaries indicates a very slight tendency 
for higher average teacher salaries to be associated with 
more third graders below the cut score, while the -.21 
with P/T (pupils per teacher) indicates that schools with 
more poor readers have slightly fewer pupils per teacher. 
Let's look at those last two 'relationships more 
carefully. 



Table 4 

Correlations Among Schools 





Read3 


ReadS 


Lowlnc 


Mlnrty 


TchSal 


P/T 


Read3 


1.00 


.86 


.69 


.72 


.13 


-.21 


Read 5 


.86 


1.00 


.74 


.74 


.16 


-.19 


Lowlnc 


.69 


.74 


1.00 


.68 


.00 


-.16 


Mlnrty 


.72 


.74 


.68 


1.00 


.30 


-.21 


TchSal 


.13 


.16 


.00 


.30 


1.00 


-.16 


P/T 


-.21 


-.19 


-.16 


-.21 


-.16 


1.00 



N • 1606 Elementary Schools 



The average elementary school in Pennsylvania has 20 
pupils for each teacher in the school. But the variation 
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in P/T is large. Five percent of the schools have a P/T 
less than 15, and in the top five percent, the P/T is 
greater than 26. But this variable exhibits very low 
correlations with other characteristics of the schools 
(except, of course, costs per pupil). The slight 
tendencies that do exist (the P/T column in Table 4) 
indicate that lower P/T are slightly associated with 
schools that serve more low income homes, in part because 
the low SES schools have extra teachers supported by 
compensatory education funds. So if smaller P/T implies 
a more effective school, that does not explain away the 
strong home effect of Figure 2. In fact it tends to work 
against it. Students from higher SES homes tended to 
attend schools with slightly more pupils per teacher, as 
seen in Table 4. These results do not refute the 
possibility that some kinds of instruction with some 
kinds of students would be facilitated by smaller 
classes, but the results do indicate that low SES schools 
are not handicapped because their classes tend to be 
larger than in high SES schools. That is clearly not the 
case. 

Turning to teacher salaries, the range in average 
salaries for the elementary schools in 1989 was from 
$18,900 to $47,000, with the average school paying an 
average salary of $30,000. So there is about a $28,000 
difference between the lowest paying school and the 
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highest. But this average teacher salary correlates zero 
(that's 0.00!) with percent low income students. So the 
big effect we find for the home is not because schools 
serving high SES homes pay their teachers more. 

It is possible that higher salaries or lower P/T 
ratios might at least explain improvement in achievement 
test performance, even though they are unrelated to SES. 
So far my many attempts at finding a significant salary 
or P/T effect have failed. The variation in P/T or 
average salary among these 1,505 schools does not explain 
the variation in percent below the cut-score, controlling 
for other possible factors, including prior performance. 
Although a previous PEPS study (Cooley, 1990) reported a 
slight teacher salary effect at the district level, it 
seems to wash out when examined at the school level. 

The dynamic that leads one to expect a connection 
between teacher salaries and student achievement goes 
like this: higher salaries attracts a larger, more 
qualified pool of applicants; if a district pays 
teachers more, they will also be able to retain their 
better teachers; better teachers will produce better 
students. One place where this may be breaking down is 
in establishing what is a "better" applicant. I was 
recently talking with a group of district superintendents 
about how they selected which teacher to hire from a 
large pool of applicants. The responses had more to do 
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with their Board's preferences for demographics than 
anything else. For example, one superintendent reported 
that out of 150 applicants for one position, he would 
have to select from among those few teacher applicants 
who actually lived in that school district. Even if they 
could hire based upon some quality indicators, they did 
not seem to have any criteria for distinguishing among 
teachers in terms of expected teacher effectiveness. 
Also, teachers who are good at integrating art into the 
curriculum, for example, may not be good at figuring out 
how to teach basic skills to low SES students. 
Summary and Conclusions 

Five years of TELLS results reveals a very stable 
set of relationships among schools and between school 
achievement and other factors, particularly factors 
associated with the homes being served by a school. It 
is also clear that there are many schools in which a 
majority of the students are not learning those basic 
skills that are needed to profit from subsequent 
schooling, or needed for eraployabi] ity. For example, 
there are about 150 schools in Pennsylvania in which, 
year after year, over half of the students score below 
minimum competence. 

The persistent, dominant importance of home factors 
has been established and re-established for at least the 
past 25 years. It is not exactly news. The initial 

d 1 ft 

ERIC ' ° 



16 

individual differences which children bring to school 
seen to determine the achievement differences among 
schools in the early grades, and those differences 
persist. So what is to be done? 

There is no reason to be optimistic about some of 
the quick fixes often proposed; for example, better 
tests, higher teacher salaries, or smaller classes. 
Teacher salary differences and pupil/teacher ratios are 
the main differences among districts with different 
spending levels, so if what districts do when they have 
more money is raise salaries or hire more teachers, then 
more money alone will probably not raise the basic skills 
levels of these low SES schools, unless many other things 
are changed as well, such as the kinds of teacher quality 
information districts have and use when they hire new 
teachers . 

What has impressed me the most as I have worked with 
these state-wide data is how interdependent the many 
variables seem to be. We are truly dealing with a 
massive system. Attempts at trying to manipulate one 
aspect of the system externally (e.g. establishing 
minimum teacher salaries) may not have the intended 
effect. Systems often react to intrusions in counter- 
intuitive ways. 

There have been several proposals for improving the 
basic skills of students from low SES homes: (1) find 
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ways to improve home processes that are relevant to 
school learning; (2) invest in preschool experiences 
that increase school readiness skills and predispositions 
for low SES students; (3) figure out how to make schools 
more adaptive to the differences which students bring to 
school. But evaluating those proposals goes beyond the 
scope of this paper. We have looked at the pre-school 
effect, and it is clearly a positive influence. James 
Bernauer will be reporting on this in a forthcoming PEPS 
report . 

What can be said with confidence about systems is 
that manipulations of single variables may not produce 
the desired effects. Systems do not change that easily 
or in easily predicted ways. Some argue that we need to 
modify the power relationships within the system before 
we can expect significant improvements. Others want to 
apply principles from the marketplace to force school 
improvements or close them down. One thing that seems 
clear is that we need to learn nore from our educational 
research efforts if we are to uffectively guide the many 
reform efforts being proposed. This modest PEPS effort 
is working toward that end. 
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